Papers with rank correlation
Correlation Coefficients and Semantic Textual Similarity (N19-1)
Copied to clipboard
| Challenge: | Existing research into semantic textual similarity has focused on word embeddings . little attention has been devoted to similarity measures between word embeds - a new study shows . |
| Approach: | They show that cosine similarity is essentially equivalent to the Pearson correlation coefficient for all common word vectors. |
| Outcome: | The proposed model outperforms the existing model on word-level and sentence-level similarity benchmarks. |
Detecting Non-Membership in LLM Training Data via Rank Correlations (2026.eacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are trained on increasingly vast and opaque text corpora. |
| Approach: | They propose a test that detects dataset-level non-membership using only grey-box access to model logits. |
| Outcome: | The proposed test detects dataset-level non-membership using only grey-box access to model logits. |
Evaluating LLMs’ Capability to Identify Lexical Semantic Equivalence: Probing with the Word-in-Context Task (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods to evaluate the capability of large language models to identify lexical semantic equivalence are not currently being used. |
| Approach: | They propose to use the Word-in-Context (WiC) task to determine whether the meanings of a target word remain identical across different contexts to evaluate their capability. |
| Outcome: | The proposed method outperforms other LLMs in the Word-in-Context (WiC) task. |
Easy to Decide, Hard to Agree: Reducing Disagreements Between Saliency Methods (2023.findings-acl)
Copied to clipboard
| Challenge: | A popular approach to unveiling the black box of neural NLP models is to leverage saliency methods, which assign scalar importance scores to each input component. |
| Approach: | They propose to use saliency methods to evaluate whether an explanation is faithful and argue that Pearson-r is a better-suited alternative to rank correlation. |
| Outcome: | The proposed methods exhibit weak rank correlations even when applied to the same model instance and advocated for alternative diagnostic methods. |
Comparing human and language models sentence processing difficulties on complex structures (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) that converse with humans are a reality, but do LLMs experience human-like processing difficulties? |
| Approach: | They systematically compare human and LLM sentence comprehension across seven challenging linguistic structures. |
| Outcome: | The proposed model achieves near perfect accuracy on non-GP structures, but struggles on GP structures. |